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SUMMARY 


Within the Intelligence Community, the term “Bayes” has been 
bandied about from time to time over the past decade, as one of a number 
of techniques to supplement traditional unalysis. There have been frequent 
assessments of the Bayesian technique and infrequent applications of it to 
the analysis of an intelligence question, usually on an experimental basis. 
During 1974 and 1975, the Office of Political Research used Bayesian analy- 
sis in three separate studics—dropping the qualifier “experimental” after 
the first—and the office intends to coztinue its application in the future, 
when such an approach is relevant. The technique produced a style of 
report which conveyed much information in a very concise format. It is our 
view and the frequently-voiced opinion of others that this represented an 
advance in communications over traditional methods of reporting. 


This handbook will attempt to explain the use of the Bayesian approach 
as a forecasting tool for intelligence. It is written for potential users of the 
technique, both to make OPR’s experience available to others, and to en- 
courage and facilitate its proper use. It is meant as a self-help guide on how 
to conduct such an exercise. Our experiences are intended to be instructive 
in a general way; the techniques described represent a particular solution to 
problems of political intelligence, but could be adapted with minor or major 
changes to other intelligence and bureaucratic situations. The presentation 
explains the rationale and format of the three OPR analyses, the procedures 
developed to administer them, and the benefits and problems of the Bayesian 
method. Neither the analyses nor this handbook wallows in methodologi«al 
soul-searching; in both cases, thc purpose is utility rather than perfection. 


There will be, of necessity, some examination of the mathematical 
formulae involved. The bulk of this handbook, however, is a catalog of the 
practical administrative details of such a project. To avoid “perfection of 
means and confusion of goals [which] seem to characterize our age” (Einstein) 
the need for careful definition of the problem and careful project design is 
repeatedly stressed, 
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BAYES 


1. What are its Capabilities arid Benefits? 


The task of intelligence is to answer questions about the capabilities 
and intentions of various international actors; information may be abundant 
or meager, but it is usually incomplete or incoherent. The main feature of 
Bayesian analysis is that it provides a rational, consistent, and objective 
process whereby many apparently unrelated facts can be combined to 
produce an overall assessment. If an intelligence problem can be formulated 
into a set of questions which can be answered in probabilistic terms—how 
likely one of a set of outcomes is—it may lend itself to Bayesian analysis. 


There is no magic and no inherent wisdom in Bayes. In simplest terms, 
the Bayesian technique consists of a statistical formula and a procedure for 
its use. (Statistics is a discipline which allows one to deal with uncertainty 
in an organized fashion without being vague or imprecise; Bayesian statistics 
differ from classical statistics in that they allow an analyst to use his own 
expert understanding of a situation along with probabilistic judgments 
based on evidence.) The Bayesian technique is an organizing device which 
allows an analyst to assign probabilities to the likelihood of various carefully 
drawn scviarios about an intelligence problem, and thereafter to evaluate 
fragments ef evidence in terms of those hypothetical scenarios. The Bayesian 
formula then aggregates these numbers mathemati sally, rather than by the 
inductive logic of an analyst, into an overall set of probabilities. This has 
several advantages: 


—More information can be extracted from the available data because 
the technique allows cach piece of evidence, central or marginal, to add its 
weight to the final assessment in a systematic way; thus, a number of small 
items can outweigh a large one, and the probabilities are not at the merey 
of the most recent or most visible item. 


—The procedure provides a reproducible sequence of steps for arriving 
at the final figures; a disagreement among analysts can thus often be seen 
to be a disagreement over the meaning of certain items of evidence rather 
than an unresolvable difference of opinion. 

--The formulation of the questions forces the analyst to consider 


alternative explanations of the facts he sees, thus loosening the bonds of 
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established opinions. In other words, he is asked to look at how well the 
: evidence explains possible scenarios other than the one he has already 
decided is most likely. 


-—The use of quantified judgments allows the results of the analysis 
to be displayed on a numerical seale, rather than through the use of terms 
like “probable,” “likely.” “unlikely,” or “possible.” Also, the work of a 
number of analysts can be arrayed in graphic form, with ranges and averages. 


—The formal procedure has been shown to be less conservative than 
analysts’ informal opinions, and to drive probabilities away from 50-50 
faster and farther than the analysts’ overall subjective judgments do. This 
is often initially unsettling for the analysts, but most have admitted that 
they later agreed with the assessment. 


—The mere fact that a team of experts is asked to assess periodically 
the evidence on an important intelligence question provides managers of 
intelligence production with a degree of assurance that the question is indeed 
being monitored. 


Before embarking on their initial Bayesian project, the OPR analysts 
assigned to coordinate the exercise reviewed the Ageney's previous experi- 
ments with Bayesian analysis, read the available literature on the theory 
and application of Bayes, and consulted with individuals familiar with the 
technique both in and outside of the A 


STAT 


The rule of Bayes is an established statistical technique with a variety 
of applications in the social sciences and industry, but its application to 
intelligence questions is both more complex and less precise. The political, 
economic, strategic, and social events of the world are imperfeetly under- 
stood and difficult to measure. Hence, to use a technique like Bayes it is 
necessary to turn to expert judgments expressed quantitatively. The values 
assigned by expert analysts are, of course, approximate, but they provide a 
rough basis for comparison and analysis. OPR’s experience suggests that it 
is relatively easy to induce analysts accustomed to qualitative expressions of 
probability to shift to numerical assessments. An ever-present danger, 
however, is the tendency to attribute more precision to the numbers than is 
warranted, and it should be stressed that the numbers are always only 
approximations, 


2. When is it Useful2 


The starting point for any investigation must always be the careful 
formulation of a relevant question. Before discussing the rule of Bayes 
further or the procedures developed to work with it, it is important to 
remember several overriding substantive considerations. 
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What is the nature of the intelligence problem; what ts the question and 
: what data are available? In the intelligence pr .cess the nature of the problem 
must dictate which methods and upproaches will be most successful. There 
ure many techniques for enhancing traditional analysis, and it is possible 
that something other than Bayes would be more appropriate to the level 
and complexity of the issue involved: trend analysis, an econometric model, 
or 2 Delphi exercise might, for example, be more relevant. Many of the more 
rigorous techniques require good data, and to the extent that a problem 
becomes more data-rich, it may actually cease to be an intelligence prcblem. 
There is also a certain delay involved in the genesis of a project, which 
might preclude its use in fast-developing situations. Tu be the type of 
question which is susceptible to Bayesian unalvsis: 


—It must lend itself to formulation in categories which overlap very 
little, such as war versus no war, or development of a complete nuclear 
capability versus development of a peaceful nuclear capability versus no 
nuclear development. Bayes is useful only when the question is expressed 
as a specific set of outcomes: thus, the Bayesian approach would be useless 
as a predictor of something as amornhous as future Middle East relations. 
The question would have to be re-cast in terms of specific alternatives, that 
is, a sect of more or less mutually exclusive scenarios of Middle East develop- 
ments. In this process, however, there is a danger that the question will be 
so simplified that the answer would be neither relevant nor interesting. 


There must be a fairly rich flow of data at least peripherally related 
io the question. For example, in the nuclear example above, data on all 
related materials and processes would be relevant. If data are sparse, the 
technique is very sensitive to each piece and is less reliable although it may 
still be useful. 


~~It must be the type of activity which produces preliminary signs and 
isnot achance or random event. For example, it would be fruitless to attempt 
to predict whether Giseard would be killed in an automobile accident. 
Bayesian analysis can measure only preparations for and indications of the 
hypothesized outcomes, 


For whom and at what level is the question zm portant? This will have a 
direct bearing on the level of resources which can be commanded for an 
analytical effort. The consumer receives a product which displays results 
concisely and graphically, and unless he or she reacts negatively to such 
displays, this can be an important consideration. 


Each production office would have to weigh the benefits and costs of 
the method—on one side the rigor, reproducibility, and sex appeal of the 
method, and on the other the need for education, monitoring, and coordina- 
tion of a number of analysts who may not find the method immediately 
congenial. Having considered the above questions, one should ask whether 
Bayes is applicable. 
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HOW TOC INITIATE A BAYESIAN EXERCISE 


1. Flowchart of Setup Procedure 
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2. Defining the Scenarios 


Once a question has been selected or assigned as a candidate for a 
Bayesian analysis, the first step is to formulate it into categories which are 
more or less mutually exclusive, that is, which cover the relevant possibilities 
and overlap as little as possible. It is imperative not to let the question run 
away with the formulation; that is, it must not be chosen simply because it 
can be answered using the Bayesian technique. If so, it may not necessarily 
be the question that the customer really wants answered. 

This formulation is easiest when there is a single event which one is 
interested in predicting. For example, the actual formulation of the first 
OPR study, in what are termed “scenarios” or ‘hypotheses,’ was: 

A. The North Vietnamese will launch a major military offensive 
(defined as a countrywide offensive on the scale of the Tet 1968 or March 
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1972 fighting. or attacks on a similar level generally confined to one or two 
military regions) before the end of June, 1974. 


B. The North Vietnamese wi!l not launch such an offensive. 


When this formulation as a dichotomy can be done, it allows a simplified 
assignment and eumputation technique, as explained later. 


Often, however, the problem is not so simple, and to formulate an intelli- 
gence problem as an either-or question would be to strip it of its complexity 
and interest. For example, the second project was designed to monitor the 
Sino-Seviet conflict, and the following possible scenarios were identified: 


A. The Soviets will undertake a nuclear strike against Chinese 
strategic or nuclear targets within six months. 


B. The Soviets will launch a large-scale conventional attack against 
China within six months. 


C. The Soviets will launch a Icealized cross-border attack, with 
limited objectives, on a scale larger than the 1969 incidents within six 
months, 


D. The Chinese will launch a localized cross-border attack, with 
limited objectives, on a scale larger than the 1969 incidents within six 
months. 


KE. One or more minority groups on either side of the border will 
revolt, follewing instigation by the Opposite side within six months. 


i’. Neither side will undertake any of the ubove types of hostilities 
within six months. 


3. Participants 


The second step is to decide the number and variety of participants 
that are needed to carry out the Bayesian analysis. OPR has been able to 
enlist participants in its various Bayesian projects from a wide variety of 
offices within CIA, analysts v-ho concentrate on economic, strategie, politi- 
‘al, and geographie affairs, or who analyze propaganda or imagery, or who 
follow the broad spectrum of current events. For Intelligence Community 
projects, participants have been enlisted from the Army, Navy, Air Force, 
DIA. NSA, INR, the Intelligence Community Staff, and the National 
Intelligence Offices. As stated before, the importance of the question will 
affect the number of persons willing or even available to give time to this 
type of effort. The number of parucipants in our reports has varied from 
six to thirteen, and although a larger number would clog the logistica] 
mechanisms and clutter certain types of graphs, there is no theoretical 
limit. Nor is there any reason the analysis could not be done by one person, 


Depending on the purpose of the exercise, itis possible for an individval, 
an office, an ageney. or an interagency group to carry out a Bayesian analv- 
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sis. OPR, in its projects thus far, has had the benefit of working on topies 
of high-level interest, with a specific consumer demand that this technique 
be applied. On a topie of lesser moment, or where there is little visible en- 
couragement from above, it would still be practical and quite useful for a 
small group or even an individual to undertake a Bayesian analysis. The 
scale on which OPR has been able to carry out its projects is not essential 
to a useful application of the technique. 


Monitoring an intelligence situation is most frequently done by an 
office or an individual within an office, and the Bayesian technique can be 
applied just as profitably on that level, without the overlay of logistics and 
cartography. In a scaled-down procedure, the logistics of the exercise would 
diminish to the puint where everything but the selection and evaluation of 
evidence could be recorded. stored, processed, und printed via a computer 
terminal. A member of OPR's could provide 
assistance in setting up the exercise an training the participants, after which 
another individual could administer the project. A smal! in-office Bayes 
could serve well to support or supplement conventional analysis, and would 
be better suited to following a fast-moving situation than the elaborate 
productions OPR has designed. It might even serve as the basis for a later 
full-scale project. 


After the exercise has been initiated, a single analyst or a small number 
of analysts in a single office could draw up a list of individual items of 
evidence; probabilities could then be assigned to each item and consolidated 
by computer or by hand. This could be done on any schedule: weekly, daily, 
or whenever new evidence appears. Readouts of revised probabilities would 
be immediately available at any time, and the results could be circulated via 
narratives, hand-drawn graphs, or computer-drawn graphs. 


4. Time Span and Frequency 


The third step is to determine how long the exercise should run and how 
often reports should be issued. This depends on the volume of data and the 
expected speed of development. For a data-rich question of great urgency 
and some probability, it may be necessary to run the exercise once or twice 
a week over a period of many months. Lacking any of these attributes, 
however, the problem may lend itself to a more leisureiy schedule. OPR's 
reports have been issued at intervals of one week, two weeks, four weeks, 
and six weeks, and have lasted for periods of six months to one year. The 
reporting period can be changed in the course of an exercise, as the question 
becomes more or less urgent, without upsetting continuity. 


Another consideration which has great practical impact upon frequency 
is the cost, in man-hours of analysis and coordination, of running & large 
project. As developed by OPR, these studies have kept to a minimum the 
time required of analysts, though this was achieved by a large expenditure 
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of time cn the part of the coordinators. Whereas each coordinator spent 
approximately two full days for each periodic assessment, cach analyst 
spent only one-quarter to one hour. Even this large investment of coordina- 
tion time might be decreased by a greater use of computer terminals and an 
interactive data-gathering program, already developed but untested. 


5. Designing the Format 


Finally, the format for presenting the results should be designed. If the 
exercise is of sufficient import to be published, that is, if there is a known or 
perceived consumer demand for it, then the design of che graphic presenta- 
tion is a very important step in communicating the analysis. Any carto- 
graphic aid, either for initial design or for continuing support, should be 
enlisted early in this process. If there is no perceivea high-level demand 
for such an analysis, meaning that it would be done for the benefit of 
working-level analysts, a less formal and expensive presentation would 
probably be more appropriate ur! just as effective. 


6. An Initial Meeting 


It will be necessary to have an initial meeting of participants at which 
the rationale, format. and procedures are explained. It is also convenient 
at chis time to collect the following information, some of which will be used 
for the list of participants in the publieation: 


— Name as it is to appear in print; 
—-Title and office name: 
Number of vears in office: 
-~Number of vears on subject; 
-—Offiee telephone number(s); 
-—Tube station or LDX address: 
—Analternats analyst to stand in during vacations; 
Starting probabilities for each hypothesis. 


The latter numbers should represent the participants’ current overall 
fecling for the probability that each event will occur as stated. (See the 
section on calculations fora further description of the starting probabilities. | 


7. Practice Runs 


It will be useful to arrange one and perhaps two practice runs, to 
familiarize the analysts with the routine, and also to work out the logistics, 
the cartography, and the printing of the report. The wording of the hypoth- 
eses should be reviewed at this point, to assure that the proper question is 
being addressed. Although the basic design of the exercise and the formula- 
tion of the questions will remain fairly constant once the formal publication 
begins, the work of familiarizing analysts with the routine will never 


7 


Approved For Release 2003/05/28 : CIA-RDP86T00608R000600180019-8 


completely end, as occasionally someone will forget to cite sources or to 
return contributions on time. 


All of the foregoing constitutes the design and definition of the project. 
Careful attention at this stage is extremely important to the subsequent 
conduci of the exercise. It is also the point at which basic questions about 
goais and intentions must be resolved. Much of it can be handled by one 
person—preferably the one who will later coordinate the — - vise—in 
consultation with boih consumers and participants. 


THE OPR BAYESIAN ANALYSES 


The three Buyesian analyses run by OPR in 1974-75 relied heavily for 
their suecess on supplementary techniques relating to data-gathering, 
presentational format, and administration. The Bayesian aspects of the 
reports formed the core of the analysis, around which a number of other 
carefully-tailored aspects were draped. 


1. More than One Analyst 


One of the central features of the studies was the use of a group of 
analysts rather than a single expert. This more than anything else influenced 
the data-gathering process, the format of the publication, and the actual 
production procedure. The reasons for this decision were: 


—To bring to the exercise a range of expertise bevond the skills and 
experience of any single analyst; 


—To supply a richer mix of evidence on the questions by asking each 
analyst to contribute anything he or she considered important. Most 
political, strategic, or military intelligence problems involve such varied 
inputs as propaganda analysis, photographic interpretation, and logistic 
matters. (The evidence was then shown to all participants without 
identifving the contributors); and 


—To provide a balance of expertise in which the effects of organizational 
and individual biases are minimized. 


It is well-known that different analysts will tend to place greater 
reliance on different types and sources of intelligenee. The consolidaicd list 
provided an opportunity for cach analyst to call an item of evidence to tne 
attention of his colleagues, and assured that the participants would coasider 
each item explicitly before judging its relevance. 


To avoid the unpredictable and oft-dccried effects of group dynamics, 
however, cach analyst worked on the probabilistic assessments individually, 
and relayed them to the OFR coordinator. This also avoided the time 
wasted in a group mecting and the problem of assembling the group on a 
convenient schedule. 
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2. Printing the Evidence 


Each periodic report contained the pieces of evidence identified and 
used by the participants, along with only a paragraph or two on the principal 
trends during the period. No attempt was made to formulate or coordinate a 
lengthy textual analysis of the situation. This allowed the reader 


To sce the basie evidence rather than just a summary and hence to 
better understand the analysts’ assessments: 


To make his own direct assessments if he so desires, or just to keep up ' 
with the copie by viewing the evidence regularly: and 


To keep a convise record of the situation. 


3. Graphs and Visibility 


The ability to portray the results of the analysis graphically was one of 
the strongest arguments for using a quantitative method like Bayes, and the 
graphs in the publications have been well-received. In each study, the 
probability of the hypothesized event, usually “how likely are major 
hostilities within a certain time period?” was immediately visible on a 
broken line or bar chart. This conveyed muck information at a glance, 
especially with the broken-line ¢c..arts, which illustrate trends far more 
concisely aad vividly than do words. On both the broken-line and bar 
charts the .ange of estimates around the central measure showed clearly 
and concisely how much disagreement there was. 


Different formats were tried in the various reports, cach demonstrating 
certain advantages and disadvantages. Arrayed on the foldout page are 
examples of the graphs and combinations of graphs (with representative 
values) used in the various reports. As a rule, the aim was to make the 
graphs as prominent as possible, while trading off the amount of information 
against the degree of clutter, the use of color against the cartographie and 
printing constraints, and the size and placement of the graphs against the 
requirements of the overall publication. 


In figure 1, from the Vietnam study, each »nalyst was represented by a 
separate line showing his or her position in successive weeks. This extensive 
detail was possible because there was only one positive question to be an- 
swered and there were only six analysts participating. (The negative scenario 
could of course be omitted because it was the mathematical complement of 
the positive one.) The assessments were represented by six lines on a single 
graph, plus one for the average (arithmetic mean). 

For the Middle East study with four scenarios and thirteen analysts, 
this format was reduced to four charts—one for each scenario—each of 
which displayed the median, the interquartile range, and the range, because 
the full representation would have been confusing, time-consuming, and 
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expensive in printing. One of those four charts is represented in figure 2. 
This format does have the disadvantage, however, that individuals are nci 
identified; consequently a full-page supplementary graph was supplied for 
that purpose, similar to figure 3b, on which each analyst gave a hunch 
judgment of hostilities in the next ninety days. This supplementary graphic 
was totally non-Bayesian. The provision of individual lines allowed each 
analyst to be visibly identified, as in the Vietnam reports, and thus to 
advocate a certain position if desired. This visibility was pleasing to the 
analysts and interesting for the readers, with the result that there was little 
if any pressure on the analysts to try to manipulate their positions on the 
other graphs (see the section on manipulation). 


In figure 3a, the assessments of thirteen analysts for six scenarios of 
Sino-Soviet hostilities were condensed into a bar graph which showed, for 
c.ch scenario, the average of the thirteen probabi'ities along with the lowest 
and the highest values assigned. To provide continuity from period to 
period and to illustrate trends, a single broken-line chart was added which 
showed the progression of the average figures over time. In addition, in- 
dividual lines were provided on a single supplementary chart, figure 3b, 
which was an attempt to. quantify and measure a related problem, the 
“level of tension” between the two countries: this was also a non-Bavesian 
assessment. Each analyst was asked to gauge his overall feeling of the level 
of tension between the two countrics on a pre-arranged historical scale, 
independently of the chances for hostilities. Gossamer though this coneept 
is, the graph performed well as an indicator of changes in the atmosphere 
between the two countries; once each individual chose his or her position on 
the chart—as a hawk, a dove, or a middle-of-the-roader—the ups and downs 
of the graph showed clear trends from month to month, and there was no 
need for the analysts to agree on a single number, 


4. Identifying the Participants 


The final presentational technique was the listing of all participants by 
name, and their identification on at least one graph with a trend line of 
ussessment. This visibility, as mentioned, helped to motivate participants, 
and in the latter two studies provided them a direct medium for expressing 
themselves in accord with their opinions on the issue. 
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Figure 1 BAYESIAN ANALYSIS ON THE LIKELIHOOD 
OF A MAJOR NORTH VIETNAMESE MILITARY OFFENSIVE 


10 December 1973-30 June 1974 
Current status of the Assessment 


Probability % 
100 | 
i 


18 27 
Participants: 


STAT 


—= Central Tendency of the Grov.n’s Estimates 


Figue2 Bayesian Analysis on the Likelihood of Major Hostilities 
in th.c Middle East in the Next 30 Days 
Assessment as of 9 April 
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Figure 3a 
A Bayesian Analysis of the Likelihood of Sino-Soviet Hostilities 
Before 1September 1974 
Current Status of the Assessment -- as of 1 March 
Probability % 
Qo 10 20 30 40 50 60 70 89) 90.100 

A The Soviets will undertake a | 

nuclear strike against Chinese ! 

strategic or nuclear targets. : 
B_ The Soviets will launch a large- L 

scale conventional attack against 

China. 


C The Soviets will launch a localized 
cross-border attack, with limited 
objectives, on a scale larger than | 
the 1969 incidents. 

D The Chinese will launch a localized | 
cross-border attack, with limited 
objectives, on a scale larger than : 
the 1969 incidents. 


either side of the border will revolt, 
following instigation by the opposite 
side. 


E One or more minority groups on L 


F Neither side will undertake any of 
the above types of major hostilities 
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Figure 3b 
Individual Assessments Of Sino-Soviet Tensions 
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HOW TO MANAGE AN ON-GOING BAYES 


1. Flowchart of Periodic Procedure: 


Analvsts Coordinator Services 


Collect and Submit 
Evidence 


Consolidate 
Evidence 


Evaluate 
Items 


Compute Revised 
Probabilities 


Revise 
Graphics 
Comp-se 

Report 


Once an exercise is started, the procedure is periodic and recurring. On 
the first day of each period, each analyst submits the items of evidence he 
has seen since last round relating to the question. The submission is in the 
form of one or two sentences summarizing the item, along with +he date, 
source, and classification, e.g., 


Cartography 


Printing 


2. Routine Procedure 


Statement in the Soviet journal, Problems of the Far [ast by 
Yumjaagiyn Tsedenbal, First Secretary cf the Central Committee 
of he Mongolian People’s Revolutionary Party, alleging that 
groups of Chinese soldiers violate the Mongolian border, fell 
trees, start forest fires, and herd diseased cattle into his country. 
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licen Segre sia ila B asl sal aN i 
A 


rT 


Approved For Release 2003/05/28 : CIA-RDP86T00608R000600180019-8 


. This can best be done on preprinted forms. The choice of data is left entirely 
to the analyst. who should include anything relevant, and exclude what can 
be judged to be irre'evant. There will be some overlap and considerable 
diversity in what is submitted. 


Later the same day, the coordinaior to whom the data is sent consoli- 
dates the items, resolving differences of wording, emphasis, and meaning. 
and distributes the complete list of items to all participants. At the cvordina- 
tor’s discretion, one standard item may be added which asks the participants 
to evaluate the volume of data itself. This is explained later in the section 
on weaknesses of the method. 


By the following day, the analysts working individually should evaluate 
the items, as explained below, and return their numerical assessments to 
the coordinator. 


3. The Calculations 


There are two ways in which the Bayesian assignments and subsequent 
caiculations can be made, a basic and a simplified way. They are both based 
on the same mathematical relationships, but one works with ratios of prob- 
abilities and the other with probabilities directly. The simplified is restricted 
to use with questions formulated in an either-or fashion, such as war or no 
war. The basic method, however, can be used in all causes, for either-or 
questions and for multiple scenario questions. Both methods require a set of 
starting probabilities which represc t each analyst's best judgment of the 
situation at the start of the exercise, and the subsequent evaluations build 
on these. 


A. The Basic Method 
For example, using the basie method and the following scenarios, 


A: Large-scale aggression by country X within six months 
B: Small-scale provocations by country X within six months 
C: No planned hostilities within six months 


an analyst might assign a set of starting probabilities to A, B, and C of 
25%, 25%, and 50% likely at the start of the exercise. (The assignment of 
these probabilities is dune entirely subjectively, although techniques exist 
to refine what a person really means by these numbers.) Note that the sum 
of these probabilities equals 100, because the scenarios supposedly cover all 
possible events. This is the only time the analyst will have to assign numbers 
which totai 100. 


Ju the course of the exercise, the analysts will evaluate each item of 
evidence as it relates to the scenarios, formulating the question as “how 
likely (0-100%) is it that thie item would oceur if A is planned?”’, “how 
likely is it that this item would occur if B is planned?”, and “how likely is it 
that this item would occur if C is planned?” The peculiar formulation of the 
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quest'ons is dictated by the nature of Bayes’ theorem, which depends upon 
conditional probabilities as explained in the first appendix. The set of 
probabilities assigned for each item need not add to 100. in this case. the 
wording of a very bellicose speech by the leader of country X might be 90% 
likely if that country were contemplating outright aggression against its 
neighbor, 90% likely if small-scale border provocations were planned, and 
80% likely even if nothing were planned. 


This procedure is followed for each item independently, without refer- 
ence to the other items. The only exception is the discretionary fina! item, 
which refers explicitly to the other items: ‘How likely is it during this period 
that this volume of information (ao more and no less) would oceur if A is 
planned?) ot cetera, 


The mathematies of revising the estimates, once the coordinator receives 
them. is straightforward though tedious. It is recommended that an inter- 
active computer program be used to save time and reduce arithmetic errors. 
The caleulations are performed for each anclyst separately; in other words, 
analyst A's set of probabilities from the previous round are updated by the 
conditional probabilities for cach item in turn, resulting ina final set of 
revised probabilities for this round. which are used for the publeation and as 
starting probabilities for the next round. The same procedure is then followed 
for each other analyst. 


The general formula for determining the revised probability for euch 
new item is 


P(S;) x PCS): 


P(S\'D) = 


eres: x PO/S;1) 
where 1 I 


P(Si)—is the starting probability for scenario i 
P(I;Si)—is the conditional probability of an item 
P(S;\/1)—is the revised probability of scenario i 
m—is the number of seenarios 


For a full elaboration of the mathematics, see Appendix 1; Appendix 2 
contains the code for interactive programs in APL and BASIC which will 
perform these calculations. The following is an example of the interaction 
using the starting probabilities assigned above, the probabilities assigned to 
the hypothetical speech, and two other sets of probabilities chosen for 
illustrative purposes. (The user’s entries start in column 5, the computer's 
responses in column 1): 


BAYES 


THIE PROGRAM CALCULATES EVENT PROBABILITIES BY THE 
BAYESIAN FORMULA. YOU WILL BE ASKED TO ENTER A SET 
OF PRIOR PROBABILITIES, THE NUMBER OF ITEMS OF EVI- 
DENCE, AND THE ITEM PROBABILITIES. 
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PRIOR PROBABILITIES FOR THIS ANALYST, IN ORDER. 
25 25 50 

HOW MANY PIECES OF EVIDENCE? 
3 

PROBABILITIES FOR ITEM 1, IN ORDER. 


90 90 80 
INTERMEDIATE RESULTS: 

26.47 26.47 47.06 (a very small change) 
PROBABILITIES FOR ITEM 2, IN ORDER. 

90 90 90 
INTERMEDIATE RESULTS: 

26.47 26.47 47.06 (no change) 
PROBABILITIES FOR ITEM 3, IN ORDER. 

60 20 10 
INTERMEDIATE RESULTS: 

61.36 20.45 18.18 (a significant change) 


P(S;/T) =61.36 20.45 18.18 


It should be noted from perusing the above, as one of the lessons to be 
learned early about the diagnosticity of evidence, that the items which have 
the greatest impact upon the final values are the items with the greatest 
spread of numbers. Item 2 had no effect on the probability of the scenario 
-because, although it had large probabilities associated with it, the item gave 
no indication of what scenario was more likely; it was not diagnostic. The 
assignment of three nineties implied that the item was very likely to occur 
no matier what scenario was true. Item 3, however, was a sensitive indicator, 
because it was very unlikely if the second and third scenarios were true; the 
effect of item 3 was to depress significantly the revised probability for those 
two scenarios. This concept of diagnosticity can usually be conveyed to 
the participants by showing them the results of the calculations for a 
round or two. 


B. The Simplified Method 


Using the simnlified method when the question is a yes-no either-or 
formulation reduces the effort for both the analyst and the coordinator. It 
is even feasible to perform all the calculations by hand. In this procedure, 
each item is evaluated as before and assigned two numbers—one for each 
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hypothesis—but they are reported as a ratio, of one over the other. Identical 
results are obtained whichever hypothesis is in the numerator, but the 
practice must of course be consistent. For example, if an item is only 10% 
likely if A is true, and 90% likely if B is true, and the ratio is to be reported 
us A over B, the evaluation for the item would be 10/90. 


The assignment procedure is followed for each item independently, 
and the mathematics of revision is then considerably simplified. For each 
analyst the starting probabilities are formed into a ratio (A—25% and 
B—75% would become 25/75) and simply multiplied against all the item 
ratios consecutively. Since multiplication is commutative, this can be done 
in any order, and much cancellation can usually be done if the whole set of 
ratios is considered. For example, 


7 96 SO 26 3H 50 3750 15 
— x — Se oe 


XX eK XK 


25 ST A OF 10 pe 250 | 


The resulting figure is the revised ratio, and should be kept as the starting 
ratio for the next round. 


The ratio must be converted to probabilities (percentages) by the follow- 
ing simple method for reporting purposes: add the numerator and denomina- 
tor together and divide the total into 100. In the case above this produces 
100/16, or 6.25. The result of this division can be used to multiply both the 
numerator and the denominator, producing the percentages for A and B, 
respectively, Again in this case, A =15 x 6.25 = 93.75, and B=1 x 6.25 =6.25. 
Remember that for the next round the starting number is the ratio, not the 
percentages. 


This procedure car be further simplified by reducing the numerica] 
scale. If probabilities are only allowed to be multiples of 10, such as 10, 
20, 50, and 90, the calculations become much easier. This level of precision 
is thought by some to be as much as an untrained mind can estimate, al- 
though many analysts have demonstrated their ability to use the full 
100-point scale. Also, because 10/90 is the same as 1/9, the ratios ean be 
ussigned using just the numbers from | through 9, 


Whichever method is used, the basic or the simplified, each analyst 
will end up with a set of probabilities, which can be reported in a number of 
ways, us described in the section on graphics. 


4. Problems in Mid-course 


The most frequent problem in conducting a Bayesian exercise is one 
common to any group project: the Management of a group of individuals, 
sume of whom will make unusual demands and require special attention. 
There will also be occasional logistics problems, and delays in cartography 
and printing, for which there is no ready solution. 
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Holidays, vacations, and dropouts are another problem which should 
be anticipated. Ideally each participant should designate an alternate who 
would be familiar with the exercise and could take over when necessary. 
When that is impractical, the exereise can be run with a diminished number 
of participants: in that case, the individual's line can cither be left out or 
extended without change, and any averages can of course be calculated on 
the basis of a smaller number of analysts. 


Finally, there is always the possibility that the basie design of the 
exercise or the formulation of the question will be found lacking, but that is 
the reason a practice run or two is recommended. 


5. Terminating and Evaluating a Bayes 


In some exercises the scenarios specify a fixed deadline at which time 
the event either has or has not taken place, such as the OPR report on the 
likelihood of a North Vietnamese offensive before July, 1974 (the end of uie 
dry season). It is clear that such a project would be terminated at that time, 
if not before. Certain other questions will be formulated to look at the 
probabilities of various scenarios within a continuously-moving time frame, 
such as the report on the likelihood of Sino-Soviet hostilities within six 
menths. A projeet such as this may be terminated at any time, or extended 
indefinitely. 


The Bayesian method upon completion results in an archive of evidence, 
evaluations, and predictions which lend themselves to various forms of 
evaluation; OPR has undertaken only the grossest measures of performance, 
however. A wealth of material lies untouched for rating collection systems, 
analyst bias, the use of intelligence in tre community, or—that most dubious 
measurement of a’l—analyst performance. 


The main criterion for evaluation is the accuracy of prediction, by 
individuals and by group, although this may not be as straightforward as it 
seems. Because of the myriad variables in the prediction equation, an event 
may oceur which was only 10% probable the day before, or an event which 
was scheduled to take place may fail to materialize. Thus there have been 
times of great uncertainty during our reporting periods when the probability 
of a scenario rose, only to full back again. Did this mean that the high prob- 
ability of the event occurring was somehow in error? In retrospect, it seems 
to mean that the event could very well have occurred at that time if other 
factors had coincided; the evaluation really cannot be considered “wrong.” 


Generally, OPR’s studies in 1974 successfully predicted non-events, 
that is, they showed that the evidence did not support any of the positive 
scenarios, and none of them occurred. In this case, the only point to be 
noted is how early in the exercise the evaluations moved away from an 
indeterminate figure toward a strong probability of no change. It has been 
our experience that the Bayesian calculations show this movement carlier 
than the analyst’s own judgment would. 
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Until one of the positive scenarios actually occurs during the course of 2 
Bayesian exercise, there is no way of knowing the predictive value of the 
technique. When the positive event does take place, it will be possible to 
conduct a much more searching evaluation of the Bayes procedure. What 
were the earliest indicators? What evidence was missing, overlooked, or 
misperceived? When did the trend lines signal a significant alteration in the 
situation? How did the Bayesian assessment compare with other intelligence 
assessments? 


INHERENT WEAKNESSES AND PROBLEMS OF THE 
BAYESIAN TECHNIQUE 


1. Limited Applicability 


The first and foremost reservation in the use of the technique, as noted 
-arlier, is that it is applicable only to certain types of questions. They must 
be capable of definition as a set of fairly distinct outcomes or hypotheses. 
Also, the procedure as developed is cumbersome enough to disecurage its 
use for questions on a crisis schedule, although it is conceivable that the 
technique could be so adapted, using computer termin:|s and = greatly 
simplified presentational techniques. 


2. Data Problems 


There is the problem of identifying which evidence is relevunt— 
whether certain peripheral items should be included. And if included, 
whether they should carry less weight than other items. The OPR exercises 
have delegated that decision to the analysts. After all, they are experts, 
and their frequent disagreement over items shows that objective measures 
of relevance would be Virtually impossible to devise. Little editorial 
judgment is imposed by the coordinator in the process of consolidating 
evidence, and any item which appears to be even peripherally relevant is 
included for evaluation. Nevertheless, each analyst is then allowed to ignore 
any item he or she considers irrelevant. This gives the participants great 
leeway over what they rate, but insures thet they at least see the evidence 
and make an explicit decision whether it is relevant. Furthermore, if two or 
more items are seen as overlapping, the participant is asked to rate only 
one of them. 

Related to this is the problem of source reliability. Although some 
methodologists have suggested that each analyst assign a numerical rating 
of source reliability along with each item, to be incorporated into the 
alculations as a weight, the OPR studies have avoided placing this extre 
burden on the analyst by requesting that the probabilities reflect how much 
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faith the analyst places in each item, As mentioned earlier, the effect of an 
item will depend on the spreed of probabilities ussigned, und items of greater 
salience and relianility will automatically be assigned a greater range of 
probabilities, if the analyst understands the process and rates items thought- 
fully. The problem of discerning items of disinformation or fabrication is 
left to the judgment of the participants: indeed. one of the benefits of the 
Bayesian process as & group exercise may be the opportunity to identify 
and sift out such ‘tems. 


The problem of negative evidence is another point which bothers strict 
methodologisis. This refers to the fact that the absence of any positive 
evidence may in itself be highly indicative, and the journalistic bias toward 
reporting events rather than non-events compounds the situation. That is, 
we tend to get news only of events or changes, whereas the fact that the 
status quo is being maintained can be quite significant. This problem has 
been recognized —as discussed in previous sections—and is partially satisfied 
by including an item such as ‘‘how likely is it that exactly this volume of 
evidence would occur (and be seen) if hypothesis A is true?"’, et cetera, 


3. Problems Over Time 


Ther? are indisputable difficulties in the use of the method ina project 
continuing over many months. First of all, the questions probably require 
some reference to a time period (explicit or implicit), that is, an event to 
secur within the preximate month or year. And as the project continues, 
the timeframe musi either contract or move forward. Contraction is illu- 
strated by OPR's first reports, which investigated the probability of an 
event before a certain fixed date. In this case, the passage of time and the 
reduction of the period remaining may itself be of significance. The 
administrator may wish to inelude an item to that effect for evaluation. 


Moviag the timeframe forward is illustrated by the second and third 
OPR studies, which looked at the probability of events “within the next X 
months.” This ‘sliding window” approach brings up the problem of retaining 
or discarding data which was evaluated months earlier with regard to an 
sarlier frame of possibilities. One solution, adopted by the second study, 
which covered a year, is to drop earlier evaluations, ma:ntaining only the 
most current three months of evidence multiplied against the original 
probabilities. The starting probabilities were also updated at intervals. 


4. Problems with Numbers 


There are also two numerical problems. The first is that multiplication 
by zero must not occur; once a probability becomes zero it can never recover. 
Thus any evaluation of zero probability must be replaced by a very smali 
number. The second is more profound, and is the problem of how individual 
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participants handle numbers, especially probabilities. It has been OPR’s 
experience that some analysts think easily in probabilities, others have to 
work at it each time, and a few need constant attention and retraining. It is 
essential that the process of assigning probabilities or ratios be reviewed on 
an individual basis when necessary. We have found that che administrator 
can recognize when a participant is uncertain of the procedure, and cne of 
the most effective teaching devices for sharpening the ability to assign 
probabilities is to show him or her the effects of each of the evaluations on 
the revised probabilities. 


5. Manipulation 


Finaliy, there is the problem of conscious manipulation. An analyst 
may assign his probabilities in a manner which reflects a pre-determined 
goal rather than unbiased judgment. This is especially true where the 
Bayesian probabilities affect the individual trend lines, as in the first study. 
Solutions to the problem could include disciplinary action or expulsion from 
the exercise, but the later OPR reports to some extent circumvented this 
problem by providing another cutlet—the supplementary graphs—for the 
expression of individual opinions. 
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APPENDIX 1 


THE STATISTICAL/MATHEMATICAL BASIS 
FOR THE CALCULATIONS 


The statistical formula which underlies this method first appeared in 
an essay by the Reverend Thomas Bayes in 1763. It is a tool of statistical 
inference, used to find the probability that an observed event was caused 
by one source rather than another. 


]. Rationale 


The purpose of Bayesian analysis is to determine the probability that 
each scenario (hypothesis) is true, given that certain items of evidence have 
been seen. The Bayesian formula allows this to be calculated in an indirect 
fashion, from the probabilitics that the items would be seen if each scenario 
were true. In our application of the method, analysts assign subjective 
values to the latter probabilities (that items would be seen), and the OPR 
coordinators calculate the former (that the scenarios are true) by applying 
the Bayesian formula. Only one other thing is necessary: a beginning set 
of probabilities for the scenarios, which the analysts supply for the initial 
round only, based solely on intuition. (This initial assignment, and the 
subsequent assessments, could be refined by various academic techniques, 
but we have not risen to that level of perfection or analyst rapport.) 


2. Derivation 


The formula itself is derived from the following basic statistical 
identities and transformations: 


S—represents a scenario, or hypothesis, such as “The USSR is 
planning to launch a nuclear attack on China.” 

I—represents an item of evidence, such as ‘‘TASS reports that the 
Chinese are deploying nuclear missiles, thereby threatening world 
peace.”’ 

P(S)—is the probability of a scenario being true. 

P(I)—is the probability of an item occurring and being seen. 
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Sn I—is the intersection of S and I, that is, both § and I, 

P(I/S)—is the conditional probability of 1 Biven S, that is, that an 
item I would occur if a Particular scenario S were true. 

P(S/I)—is the conditional Probability of § given I, that is, that a 
Scenario would be true if an item were seen. 


1. P(Sn J) =P(In S) commutativity of intersection 
P(In 8S) 
2. P(I/S) = PS) definition of conditional probability 


PSn 1) Pita S) 


3. P(I/S) = cr f 1 and 2 
(1/S) PS) PS) rom 1 an 
4. P(In 8) = P(S) x P(T/S) transposition of 2 


At this point, we define m mutually exclusive Scenarios, Si, of which one 


must occur. That is, 
m 
DPC) =1 


i-t 
5. P(I)=P(I 9 Si) +P (In Sa)+.. -+P(I n Sm) by definitions 


6. P(I)= (P(S;) x P(I/S,))+.. -+(P(Sm) x P(I/Sm)) from 4 and 5 
7. P(D= Desi) x P(I/S;)) restatement of 6 
eel 


The desired probability is the probability that each Sj; is true given that 
@ piece of evidence has been Seen, or, using this Notation, P(S,/I), 


P(In §;) 
8. P(S,/I) =——_— from 3 
P(I) 


9. P(S;/I) “Sq substitution of 4 in 8 


P(S;) x P(I/S;) 
10. P(S;/I) = TO Substitution of 7in 9 


DP) « Pals,)) 
1 


1 
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The last formula, which is the rule of Bayes, says that, given an 
analyst’s starting probabilities P(S;) for all scenarios, and his assessments 
of how likely an item would be if S were true, P(I/S;), then the new 
probability of S;, now that I has occurred, can be calculated using formula 10. 


3. Example 


Assume an analyst has assigned the following probabilities, intuitively, 
to scenarios one and two, that the USSR is planning to launch a nuclear 
attack on China within six months, and that she is not: 


P(S;) =10% or .1 
P(S») =90% or .9 


Also assume that the following item comes in and that the analyst 
assigns probabilities that it would occur, first assuming that Russia is 
planning a nuclear attack, and second, assuming that she is not planning one: 

“TASS reports that the Chinese are deploying nuclear missiles, thereby 
threatening world peace.” 

P(I/S;)=99% or .99 
P(I/S2) =80% or .8 

This information can be used to revise the probabilities that each 

scenario is true by using the Bayesian formula: 


P(S1)x PUW/S;) .1x.99 ~— .099 


P(S,/I) = ite eS eee 
099+ .72 .819 
D{PGI x PUIS)) 
it 
Similarly, 
P(S2) x P(I/S2) I+.8 2 
P(S2/1)= = “= gg 


099+.72 819 


Notice that the two new probabilities add to 1 (100%) even though the 
item probabilities did not. As this is a recursive process, in which a succession 
of items of evidence are assessed, the new probabilities are used as P(S;) in 
calculating the effect of the next item. 
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APPENDIX 2 


INTERACTIVE PROGRAMS IN APL AND BASIC 
1. Interactive APL Program 


Vv B;INSTR;INSTR1 ; INTERMEDIATE ; COUNT ;PRIOR;RFVISED;TIMFS ;PROB;TOTAL 
to 


[1] 

[2] ae 

(3) "INSTRUCTIONS? TYPF YES OR NO.' 

C4J INSTR1i«3-(p( INSTR )) 

C5] +16x1INSTR1 

C6] ‘THIS PROGRAM CALCULATES EVENT PROBABILITIFS' 

(7) BY THE BAYESIAN FORMULA' 

[8] 1 1 

[9] "YOU WILL BF ASKED TO ENTFR A SFT OF PRIOR PROBABILITIFS, 
{10] 'THE NUMBFR OF ITEMS OF EVIDENCE, AND THF ITFM BROAAE EE THEA 
C11] 

[12] one CORRECT A MISTAKE: ON A DELTA, MOVF THF CURSOR BACK AND HIT 
[13] !' ON A 2741, BACKSPACFE AND HIT ATTN,! 
C14] 'IF YoU WANT TO STOP, TYPF THRE + KREY,' 

C15} ' 


[16] ‘PRINT INTFRMEDIATF RESULTS? TYPF YFS OR NO. 
[17] INTERMEDIATE+3-(p(INTERMEDIATE+ ) ) 
C18] ' 1 
19] couNnTr+o 
[20] 'PRIOR PROBABILITIFS FOR THIS ANALYST, IN ORDFR' 
[21] PRroR 
(22] REVISED+PRIOR 
[23] ‘HOW MANY PIFCES OF EVIDFNCE! 
[24] TIMFs< 
[25] CouNnT+couNT+1 
[26] 'PROBABILITIFS FOR ITEM ';COUNT;', IN ORDER’ 
(27) Pros 
[28] +31x1((pPPROB) =(pPRIOR)) 
[29] ‘WRONG NUMBER OF FNTRIES' 
. [30] +26 
[31] TOTAL++/(REVISFDxPROB ) 
(32] RFVISFD«(RFEVISEDxPROB)+?TOTAL 
C33] +35x. (INTERMEDIATE) 
C34] "INTERMEDIATE RFSULTS: ':RFEVISEDx100 
[35] +25. (COUNT<TIMFS) 
: (36] ry ' 
; (37) 'P(SilI) = 's;RFEVISEDx100 
: (38) : 
[39] 
: [40] 
: (41) 
: (42) 


ee ee 
-o «= 


- 
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2. Interactive BASIC Program 


100 DIM BC10) 

105 DIM 1¢€10) 

110 PRINT “INSTRUCTIONS? TYPE YES @R N@Be™ 

120 INPUT AS 

130 IF AS = “NO THEN 200 

140 PRINT “THES PROGRAM CALCULATES EVENT PR@BABILITIES” 
150 PRINT “RY THE BAYESIAN F@RMULA." 

160 PRINT “YG@U WILL BE ASKED T@ ENTER THE NUMBER QF" 
170 PRINT "SCENARI®S, A SET @F PRIOR PROBABILITIES»” 
180 PRINT "THE NUMBER @F ITEMS @F EVIDENCE» AND THE" 
190 PRINT “ITEM PR@BABILI TIES-" 

192 PRINT 

194 PRINT "T@ CORRECT A MISTAKE» USE THE @ KEY +" 

196 PRINT “IF Y@U WANT T@ STOP» HIT THE BREAK KEY «" 
198 PRINT 

200 PRINT "PRINT INTERMEDIATE RESULTS? TYPE YES @R NO." 
210 INPUT AS 

220 PRINT 

306 PRINT "NUMBER @F SCENARI®@S» INCLUDING NULL:" 

310 INPUT S 

320 IF S<2@ THEN 990 

330 IF S>10 THEN 990 

350 PRINT “PRI@R PROBABILITIES FOR THIS ANALYST» IN ORDER?" 
360 FOR Cl = 1 TES 

370 PRINT Cis" "3 

360 INPUT BCC1) 

390 NEXT C1 

400 PRINT “"H@W MANY PIECES @F EVIDENCE?" 

410 INPUT E 

4390 F@k C2 = 1 TOE 

440 PRINT “PR@BABILITIES FOR ITEM*sCes"IN ORDER: ** 
460 FOR C1 = 1 TES 

40 PRINT Cis*s"'s 

480 INPUT 1C(C1) 

490 NEXT C1 

500 D= 0 

510 FOR C3 = 1 T@S 

sec0 D=D+ ¢ BCC3) * 1403) 9 7 D 

530 NEXT C3 

550 FOR C3 = 1 TES 

560 BCC3) = ¢ BCC3) * 1(C3) 0 / D 

S70 NEXT C3 

630 IF AS = “NO THEN 690 

650 PRINT “INTERMEDIATE RESULTS:* 

660 F@R Cli = 1 TOS 

@O PRINT Cis*s"s BCCI) 

650 NEXT Cl 

682 PRINT 

690 NEXT C2 

710 PRINT “PCS/I)D = " 

720 FOR C1 = 1 T8S 

730 PRINT Cistts"3¢ BCCI) * 100 ) 

140 NEXT Cl 

7190 GO TO 998 

990 PRINT "T@O FEW OR TOO MANY SCENARIOS" 

998 STOP 

999 END 
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